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This  paper  presents  a  procedure  for  integrating  the  findings 
of  psychological  research  of  interest  to  the  Department  of 
Defense  (DoD).  In  recent  years,  there  has  been  increasing 
emphasis  on  applying  more  objective,  quantitative  methods  to  the 
integration  of  research  results.  This  paper  reviews  various 
approaches  to  research  integration,  describes  the  meta-analysis 
approach  of  Glass  (1977),  and  suggests  some  directions  for  the 
application  of  meta-analytic  procedures  to  military  psychological 
research  . 


Introduc  tion 


The  number  of  research  studies  has  rapidly  increased  in  many 
disciplines,  including  psychology.  Within  the  psychological 
arena  lies  a  sizable  body  of  research  of  interest  to  the 
Department  of  Defense  (DOD).  For  decades,  a  wide  variety  of 
psychological  research  has  been  conducted  in  military  settings 
and  with  military  populations  (Oliver,  in  press).  Yet  this 
proliferation  of  research  seems  not  to  have  advanced  the  state  of 
science  to  the  extent  that  one  might  have  expected.  In 
considering  the  problem  of  so  much  research  and  the  relatively 
few  conclusions  that  can  be  drawn  from  it,  Frank  Schmidt 
concluded  that  "the  most  important  problem  in  psychology  and  the 
social  sciences  today  is  the  failure  to  produce  cumulative 
knowledge"  (Schmidt,  1980). 

Accordingly,  the  purpose  of  this  paper  is  to  review  various 
approaches  to  research  integration,  to  describe  in  more  detail 
the  meta-analysis  approach  of  Glass  (1977),  and  to  suggest  some 
directions  for  the  application  of  meta-analytic  procedures 
to  military  psychological  research. 

Approaches  to  Research  Integration 

In  this  paper,  research  integration  refers  to  combining  the 
research  results  of  a  group  of  studies.  Several  procedures  that 
can  be  used  to  integrate  research  findings  are  described  below. 


The  views  expressed  in  this  paper  are  those  of  the  author  and  do 
not  necessarily  reflect  the  views  of  the  US  Army  Research 
Institute  or  the  Department  of  the  Army. 
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Literary  approach.  The  literary  or  narrative  approach  has 
been  the  traditional  procedure  for  integrating  research  findings. 
The  reviewer  reads  the  studies  on  a  given  topic  and  attempts  to 
derive  generalizations  about  the  outcomes  of  that  body  of 
research.  This  can  be  an  extraordinarily  difficult  task, 
especially  if  the  number  of  studies  is  large  or  the  results 
conflict.  In  addition,  due  to  the  subjective  nature  of  this 
process,  different  reviewers  may  reach  different  conclusions  when 
integrating  the  results  of  essentially  the  same  body  of  research 
(Smith,  Glass,  4  Miller,  1981).  Such  reviews  typically  end  with 
a  call  for  more  research  to  resolve  the  conflicts. 

Box-score  approach.  In  the  box-score  or  vote-counting 
approach,  a  directional  hypothesis  is  assumed.  The  reviewer 
classifies  the  findings  of  each  study  into  positive  significant, 
negative  significant,  and  nonsignificant  categories  and  tallies 
the  results.  (Reviewers  usually  do  not  indicate  the  direction  of 
the  nonsignificant  results.)  Again,  the  results  may  conflict  and 
make  it  difficult  for  the  reviewer  to  draw  firm  conclusions  about 
the  research  being  integrated.  One  objection  to  the  box-score 
approach  is  that  it  does  not  take  into  account  the  magnitude  of 
the  significant  results.  Hedges  and  Olkin  (1980)  have  also 
demonstrated  that  the  probability  is  high  of  failing  to  conclude 
there  is  a  positive  effect  when  in  fact  there  is  and  also  that 
the  probability  of  making  this  error  may  increase  as  the  number 
of  studies  increases. 

Meta-analysis .  According  to  Glass  (1977),  meta-analysis  is 
the  statistical  analysis  of  the  analytic  results  from  a  number  of 
independent  studies.  To  accomplish  such  an  analysis,  the 
findings  of  the  individual  studies  must  be  quantified.  The 
approach  developed  by  Glass  and  his  colleagues  (Glass,  McGaw,  4 
Smith,  1981)  is  described  in  the  next  section  of  this  paper. 
However,  there  are  other,  related  meta-analysis  procedures. 
Hunter,  Schmidt,  and  Jackson  (1982)  have  expanded  their  validity 
generalization  techniques  into  a  full-fledged  me ta-analy tic 
approach.  Hedges  and  Olkin  (  1 9 8 0 ,  in  press)  have  advanced 
statistical  theory  and  applications  to  meta-analysis.  Also, 
Rosenthal  (1978)  has  summarized  a  number  of  methods  for  combining 
the  probabilities  obtained  from  the  result  of  two  or  more 
studies . 


The  Meta-Analysis  Approach 

The  meta-analysis  procedure  most  frequently  employed  to  date 
(and  the  one  which  will  be  described  below)  has  been  the  one 
developed  by  Glass  and  his  colleagues.  The  unit  of  analysis  in 
this  approach  is  a  standardized  mean  difference  called  the  effect 
size . 


To  calculate  the  effect  size.  In  their  meta-analysis  of 
psychotherapy  research,  Smith  and  Glass  (1977)  (see  also  Smith, 
Glass,  4  Miller,  1980)  have  defined  the  effect  size  as  the 
difference  between  the  means  of  the  experimental  and  the  control 
groups  on  a  given  dependent  variable  divided  by  the  standard 


deviation  of  the  control  group.  That  is, 


Effect  Size 


V  MC 


Using  this  formula,  it  is  possible  to  express  the  standing 
of  the  average  experimental  group  subject  in  terms  of  the  control 
group  distribution  (see  Smith  &  Glass,  1977;  Spokane  &  Oliver, 
1983).  As  McGaw  and  Glass  (  1 9  8  0 )  warn,  effect  size  is  a  simple 
concept,  but  its  calculation  can  be  complicated  by  differences  in 
experimental  design  and  the  metric  used.  These  authors  suggest 
using  the  control  group  standard  deviation  as  the  denominator 
(McGaw  &  Glass,  1980,  pp.  106-123).  Hunter  et  al.  (1982)  agree 
with  Glass  that  the  experimental  group  treatment  may  affect  the 
experimental  group  standard  deviation  as  well  as  the  experimental 
group  mean.  However,  they  argue  for  using  the  within-group 
standard  deviation  because  the  control  group  standard  deviation 
has  more  sampling  error  and  also  because  research  reports  are 
likely  to  contain  values  for  _t  and  F  and  fail  to  report  the 
control  group  standard  deviation  (Hunter  et  al.,  1982,  p.  101). 

Reports  lacking  means  and  standard  deviations.  If  the  data 
(means  and  standard  deviations)  are  not  reported  in  the  study  in 
question,  it  may  be  possible  to  estimate  or  retrieve  the  required 
data  from  those  statistics  that  are  reported  (such  as  t  and  F  ). 
Procedures  are  also  available  for  calculating  effect  sizes  from 
correlation  coefficients,  nonparametric  statistics,  and 
dichotomous  outcome  variables  (Glass  et  al.,  1981). 

Occasionally,  data  can  be  obtained  from  the  author  of  the  article 
or  report,  but  this  procedure  is  time-consuming  and  generally 
unproductive  (Oliver  &  Spokane,  1983). 
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Suggestions 
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Following  are  some  suggestions  for  capitalizing  on  research 
already  accomplished  and  providing  direction  for  future  research 
in  the  DoD. 

A 

The  meta-analysis  approach  has  been  used  to  integrate 
research  findings  in  dozens  of  topical  areas,  primarily  in  the 
disciplines  of  psychology  and  education.  It  is  important  that 
such  efforts  be  continued  in  order  to  determine  what  we  can 
confidently  conclude  from  our  research  as  well  as  to  identify 
gaps  in  our  knowledge.  In  fact,  it  is  likely  that  journal 
editors  eventually  will  require  that  a  meta-analysis  approach  be 
used  for  all  literature  reviews. 

Until  now,  most  research  integration  has  concerned  the 
civilian  sector.  These  findings  will  have  their  greatest 
applicability  for  DoD  civilian  research.  Little  quantitative 
integration  seems  to  have  been  accomplished  for  research  relating 
to  military  populations  and  settings.  At  present,  we  are  not 
certain  to  what  extent  findings  based  on  civilian  populations 
apply  to  the  military.  It  will  be  important  to  use  the 
meta-analysis  approach  to  integrate  the  military  research  and 
then  to  compare  the  resulting  findings  with  those  of  research 
conducted  in  the  civilian  sector. 
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